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FOREWORD 


This  report  was  prepared  by  the  Navy  Personnel  Research  and  Development  Center 
(NAVPERSRANDCEN)  to  document  its  involvement  in  the  training  of  structured  problem 
solving  and  basic  graphic  methods  in  a  Total  Quality  Leadership  setting.  This  effort  was  funded 
by  reimbursable  work  unit  98  0WR0(X)34  under  the  sponsorship  of  Gerald  C.  Hoffmann, 
Specification  Control  Advocate  General  for  the  Assistant  Secretary  of  the  Navy  (Ship  Building 
and  Logistics)  as  part  of  his  goal  to  develop  Productivity  Gain  Sharing  (PGS)  in  Navy 
organizations.  Midway  through  the  project  the  sponsorship  was  transferred  to  the 
Undersecretar\'  of  the  Navy,  the  Honorable  J.  Daniel  Howard  as  part  of  his  effort  to  support  the 
implementation  of  PGS  in  a  total  quality  environment.  We  wish  to  express  our  sincere 
appreciation  to  both  sponsors  for  their  support  of  this  project. 

The  authors  also  wish  to  thank  CAPT.  L.  B.  Blumberg  of  the  Fleet  Combat  Direction 
Systems  Support  Activity  for  his  help  in  organizing  and  facilitating  these  training  sessions  at  his 
command. 

Questions  regarding  this  \v.ork  can  be  directed  to  Dr.  Paula  Konoske.  Code  162.  Navy 
Personnel  Research  and  Development  Center.  San  Dietio.  CA  92152-6800.  (619)  553-2944  or 
AUTOX  ON  553-7944 


THO.XIAS  F.  FINLEX' 
Captain.  I'.S.  .Navy 
Commanding  Officer 


RICHARD  C.  SORENSON 
Technical  Director  (Acting ) 


SUMMARY 


Problem 

To  improve  quality  and  productivity.  Navy  and  Marine  organizations  are  adopting  a 
management  approach  known  as  Total  Quality  Leadership  (TQL).  A  large  part  of  TQL  is  based 
on  the  use  of  statistical  measures  and  graphical  techniques  to  improve  process  and  product 
quality.  Many  Department  of  the  Navy  organizations  have  difficulty  establishing  and 
maintaining  a  systematic  approach  to  process  improvement  using  these  statistical  and  graphical 
methods. 

Purpose 

This  case  study  documents  the  effons  made  by  the  Fleet  Combat  Direction  Systems 
Suppon  Activity  (FCDSSA)  to  train  a  select  group  of  employees  to  serve  as  "facilitators"  in  the 
organization's  TQL  system.  Personnel  from  the  Navy  Personnel  Research  and  Development 
Center  (N.AVPERSR.A.NDCEN)  were  tasked  to  develop  and  deliver  a  course  on  "Structured 
Problem  Solving  and  the  Basic  Graphic  Methods"  to  a  group  of  about  20  FCDSSA  employees. 
These  20  people  will  ser\e  as  facilitators  for  future  Process  Action  Committees  (PACs)  assigned 
to  improve  different  processes  at  FCDSSA.  The  facilitators  w'ill  apply  what  they  learned  in  the 
course  to  help  guide  the  P.ACs. 

Approach 

The  personnel  from  N.AVPERSRANDCEN  designed  a  course  for  20  people  (mid-level 
managers,  supersisors.  and  non-super\'isors)  that  met  for  3  hours  per  day.  Monday  through 
Friday,  for  2  weeks.  The  format  of  the  cour.se  wa.s  a  mi.xture  of  formal  pre.sentaiions. 
discussions,  demonstrations,  and  group  exercises.  The  basic  content  of  the  course  was  structured 
around  a  real  process  (software  development  and  maintenance  of  the  Advanced  Combat 
Direction  System  (ACDS)  Block  0  program)  that  was  identified  by  the  Executive  Steering 
Committee  (ESC)  as  important  to  FtTDSS.A.  The  process  was  analyzed  using  the  Plan-Do- 
Check-.Act  (PDCA)  Cycle  (otherwi.se  known  as  the  Shewhart  Cycle  or  the  Deming  Cycle)  and 
demonstrated  the  application  of  the  seven  basic  graphic  methods  discussed  by  W.  E.  Deming  and 
others. 


Plan:  The  first  step  was  to  identify  critical  product  and  service  requirements  for  the 
major  customers,  and  develop  a  process  flow  diagram. 

Do:  The  next  step  was  to  identify  imponant  quality  variables  and  develop  quantifiable 
measures  of  these  variables.  Once  the  measures  were  developed,  the  group  designed  a 
plan  for  recording  and  collecting  data  on  these  measures. 

Check:  The  group  collected  the  data  and  used  the  graphical  me’.nods  to  display  and 
summarize  the  results. 

Act:  The  group  selected  additional  process  variables  they  believed,  based  on  knowledge 
from  the  previous  steps,  were  major  contributors  to  qua'ity.  They  then  planned  and 
carried  out  actions  to  gain  more  knowledge  and  improve  r’.ic  .ACDS  process. 


Results 


The  participants  claimed  that  the  course  helped  to  improve  communication  withsn 
FCDSSA  (especially  across  departments).  They  saw  great  value  in  applying  the  statistical  and 
graphical  techniques  to  a  real  process  rather  than  working  on  a  set  of  pre-packaged,  hypothetical 
data.  How  much  they  really  learned  from  the  course  is  unknown  at  this  point,  but  the  real  test 
will  come  when  thev  must  applv  their  knowledge  to  other  processes  being  investigated  bv  their 
PACs. 


From  the  point  of  view  of  the  designers  of  the  course  (the  NAVPERSRANDCEN 
personnel),  the  course  was  too  short.  It  was  diffi^'ult  to  see  process  improvements  over  a  2  week 
period.  Ideally,  the  course  should  meet  once  a  week  or  once  every'  2  weeks  for  several  months. 
During  the  intervals  between  meetings,  the  students  could  collect  data  on  quality  measures. 
These  data  could  then  be  summarized  over  a  sufficiently  long  period  to  allow  trends  to  emerge 
and  to  judge  the  stability  of  the  process. 

.Another  problem  with  the  course,  perhaps  related  to  its  brevity,  was  the  difficulty  in 
generating  process  measures.  The  participants  seemed  to  have  difficulty  separating  process 
measures  (which  are  intended  to  be  impersonal)  from  job  performance  measures  (which  are 
usually  intended  for  individual  performance  review's).  Perhaps  with  enough  experience  in 
generating  process  measures,  the  students  would  become  more  .skilled  at  making  this  distinction. 

Conclusions 

It  is  clear  that  the  statistical  and  graphical  tools  presented  in  these  training  materials  can 
be  applied  to  most,  if  not  all.  proces.ses.  Surely,  if  something  as  complex  and  intractable  as 
software  development  and  maintenance  can  succumb  to  these  analytic  techniques,  there  are  very 
few  processes  that  will  be  excluded.  Any  impediments  to  TQL  and  the  tools  of  TQL  are  not 
inherent  to  the  nature  of  the  process.  Rather,  the  difficulties  mostly  arise  from  managemem's 
reluctance  to  change  the  status  quo. 

Recommendations 

1.  Organizations  interested  in  TQL  should  provide  training  to  their  employees  on  the  statistical 
and  graphical  methods  for  quality  improvement.  The  course  should  be  taught  by  trained 
per.sonnel  and  should  meet  on  a  weekly  or  biweekly  basis  for  several  months. 

2.  The  statistical  and  graphical  methods  can  be  learned  effectively  by  applying  these  tools  to  a 
real  process  that  has  significance  to  the  organization.  The  Executive  Steering  Committee  (ESC), 
or  a  similar  body  of  top  managers,  should  be  responsible  for  identifying  the  process. 

.^.  The  training  should  cover  all  of  the  statistical  tools  and  graphical  methods  that  an 
organization  is  likely  to  need.  The  training  should  demonstrate  how  these  tools  and  techniqiaes 
can  be  applied  at  several  different  phases  of  the  PDCA  cycle,  and  are  not  restricted  to  only  a 
single  phase. 

4.  Training  should  be  extended  over  several  months  so  that  long  term  data  can  be  collected. 
Control  charts  should  be  used  to  detenmne  the  stability  of  the  process.  If  the  process  is  .rot 
stable,  an  attempt  should  be  made  to  stabilize  it.  if  possible,  before  applying  other  charts  and 
computations. 
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INTRODLCTION 


Background 


To  improve  quality  and  productivity.  Navy  and  Marine  organizations  are  adopting  a 
management  approach  known  as  Total  Quality  Leadership  (TQL).  This  approach  is  based  on  a 
set  of  management  principles  and  statistical  methods  that,  when  combined,  reduce  the  factors 
leading  to  poor  product  quality.  This  action,  in  turn,  increases  productivity  and  reduces  excessive 
costs  (Dockstader,  Doherty,  &  Konoske,  1989;  Houston,  Shettel-Neuber  &  Sheposh,  1986). 
Although  some  private  sector  companies,  such  as  Hewlett-Packard  and  Nashua  Corporation, 
have  demonstrated  successful  application  of  these  management  principles  and  methodologies, 
there  are  only  a  few  government  agencies  that  have  moved  ahead  with  the  same  degree  of 
commitment  (Spechler,  1988;  Walton,  1986). 


In  the  fall  of  1989.  Captain  Blumberg,  Commanding  Officer  of  the  Fleet  Combat 
Direction  Systems  Support  Activity  (FCDSSA),  San  Diego,  committed  his  organization  to 
implementing  a  TQL  approach.  FCDSS.A  requested  the  assistance  of  the  Navy  Personnel 
Research  and  Development  Center  (NAVPERSRANDCEN)  in  this  endeavor. 
N.A\’PERSR.ANDCEN  provided  FCDSSA  with  TQL  awareness  training,  implementation 
training,  and  consultation  service>.  During  FY9().  NAVPERSRANDCE.N  researchers  worked 
closely  with  FCDSS.A  management  as  implementation  efforts  began. 


•An  organizational  infrastructure  based  on  cross-functional  teams  is  basic  to 
NAVPERSRANDCEN's  TQL  implementation  model  (Houston  &  Dockstader,  1V88).  An 
Executive  Steering  Committee  (ESC)  represents  the  highest  level  of  management.  The  ESC. 
made  up  of  the  top  managers  of  the  organization,  identifies  the  strategic  goals  for  organizational 
quality  improvement  efforts.  The  ESC  charters  Quality  Management  Boards  (QMBs)  to  work  on 
significant  work  processes  within  the  organization.  Q.MBs  generally  consist  of  those  middle 
managers  responsible  for  a  specific  product  or  service.  At  FCDSS.A  there  are  three  standing 
cross-functional  QMBs:  (1 )  Department  Directors  QMB.  (2)  Chief  Programmers  QMB.  and  (3) 
Planners  QMB.  The  QMB  members  relate  the  ESC's  quality  improvement  goals  to  specific 
output.s  and  processes  uiihin  their  span  of  control.  The  indi\  idual  Q.MBs  then  organize  ad  hoc 
Process  .Action  Committees  (P.ACs).  sometimes  referred  to  as  Process  .Action  Teams  (P.ATs). 
that  collect  and  anal\  ze  information  about  work  processes.  PACs  use  basic  statistical  methods  to 
analyze  a  process  and  identify  potential  areas  for  improvement. 


The  purpose  of  this  paper  is  to  document  the  structured  problem  solving  and  basic 
graphic  methods  training  conducted  at  FCDSS.A  by  NAVPERSRANDCEN  researchers. 
Examples  from  an  actual  FCDSS.A  work  process  are  used  to  illustrate  the  problem  solving 
techniques  and  the  graphic  methods 


Organizational  ()\er\ie« 


FCDSS.A  develops  and  maintains  computer  programs  for  the  Department  of  the  Navy,  It 
provides  life  cvcle  support  for  over  50  computer  programs  installed  at  over  1.215  sites 
throughout  the  world.  Located  in  San  Diego.  FCDSS.A  has  approximately  380  military  and 
CIV  ilian  employees  w  iih  an  annual  operating  budget  of  over  30  million  dollars. 


APPROACH 


Course  Overview 

At  FCDSSA's  request,  NAVPERSRANDCEN  developed  a  course  that  introduced  the 
basic  statistical  methods.  The  course  was  designed  to  familiarize  the  studer.'^  with  structured 
problem  solving  and  the  graphic  methods. 


Twenty  employees  participated  in  the  training.  Two  instructors  conducted  a  30-hour,  (3 
hours  per  day),  course  given  over  a  2-week  period.  The  format  consisted  of  formal  presentations, 
class  discussion,  and  small  group  interactions.  The  class  discussions  were  facilitated  by  idea 
generation  techniques,  such  as  the  nominal  group  technique  (NGT)  and  brainstorming. 


Instructors  used  the  Plan-Do-Check-Act  (PDCA)  cycle,  also  known  as  the  Shewhart 
cycle  or  the  Deming  cycle,  as  the  foundation  for  the  structured  problem  solving  approach. 
Shewhart  ( 1931 )  de\  eloped  the  PDCA  cycle  as  a  way  to  apply  the  scientific  method  to  practical 
problems.  The  instructors  introduced  the  class  to  seven  graphic  methods  (flow  charts,  cau.se- 
and-effect  diagrams.  Pareto  diagrams,  histograms,  scatter  diagrams,  run  charts,  and  control 
charts).  The  course  was  designed  so  that  the  participants  could  become  familiar  with  the 
problem  solving  structure  (PDCA)  and  the  graphic  methods  as  applied  to  an  actual  FCDSSA 
process.  Table  1  outlines  the  topics  and  activities  for  the  course. 


The  actual  number  of  panicipants  cannot  be  determined  because  some  dropped  out  and 
others  were  added  after  the  course  began.  Fi\  e  of  the  participants  had  first-hand  experience  with 
the  process  analyzed  and  provided  the  group  with  valuable  knowledge.  The  other  participants 
represented  the  remaining  codes  (departments).  It  was  the  intent  that  they  would  acquire 
problem-solving  expertise  and  knowledge  about  the  graphic  methods  that  they  could  then  use  for 
team  process  improvement  effons  in  their  respective  codes.  Their  roles  as  facilitators  would  be 
collateral  duty.  All  of  the  participants  had  attended  a  4-hour  introduction  to  the  basic  concepts 
and  principles  of  TQL. 

The  course  materials  consisted  of  the  following: 

Text  Book: 

Stanstical  Methods  for  Quality  Improvement  by  H.  Kume  (198.5) 

Selected  Readings: 

Managing  With  Statistical  Methods  "  by  J,  Siegel  (1982) 

"Process  Improvement  b>  R.  Moen  and  T.  Nolan  (1987) 

"Understanding  Variation  "  by  T.  Nolan  and  L.  Provost(199()) 

Videotape:  A  Japanese  Control  Chart  narrated  by  Donald  Wheeler  (1984) 

Lesson  Viewgraphs  prepared  b\  N.AVPERSRANDCEN 


Table  1 


Course  Outline 


TOPIC 

ACTIVITY 

1. 

Overv  iew  of  TQL  Principles 

Viewgraph  presentation 

2. 

Introduction  to  PDCA  cycle 

Viewgraph  presentation 

3. 

Introduction  to  total  quality 
process  improvement 

Viewgraph  presentation 

Group  discussion 

4. 

Overv  iew  of  ACDS  Block  0  process 

Nominal  group  technique 

Group  discussion 

5. 

Customer  identification 

Product/Output  identification 

Product  quality  characteristic 
identification 

Group  discussion 

6. 

Process  flow  chan 

Deployment  flow  chart 
exercise 

7 

S'  V. technical  analysis 

L.  perations  identification 

Key  variance  identification 

Variance  matrix  generation 

S. 

Cauve-and-effect  analysis 

Cause-and-effect  diagram 
development 

9. 

Data  collection  strategies 

Group  discussion 
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Pareto  chans  and  histograms 

Group  development  and 
interpretation 

11. 

Scatter  diacrams.  run  charts, 
and  control  charts 

Group  development  and 
interpretation 

12. 

Group  data  collection  plans 

Group  discussion  and 
pre.sentations 

Note.  TQL  =  Total  Quality  Leadership.  PDC.A  =  Plan-Do-Check-Act,  ACDS  =  Advanced 
Combat  Direction  System. 


Before  the  start  of  the  course,  the  ESC  (which  consists  of  the  Commanding  Officer. 
Executive  Officer,  Technical  Director.  Plans  Officer.  Quality  Advisor,  and  Senior  Chief 
Programmer  identified  and  prioritized  customer  product  requirements.  The  fleet  customer  and 
the  sponsor  identified  the  Advanced  Combat  Direction  System  (ACDS)  Block  0  process  as 
resulting  in  an  important  product.  Thus,  the  ESC  selected  it  as  the  process  to  be  used  during  the 
training. 


Advanced  Combat  Direction  System  (ACDS)  Block  0  Process 


The  software  development  plan  for  the  ACDS  Block  0  process  begins  when  the  sponsor 
(NAVSEA)  provides  high-level  requirements  and  specifications  to  the  program  analysts  at 
FCDSSA.  For  example,  the  sponsor  might  request  a  program  that  tracks  100  airplanes 
simultaneously.  The  design  analyst  would,  in  turn,  take  those  high-level  requirements  and 
transform  them  into  lower-level  specifications  for  the  programmers.  These  lower-level 
specifications  include  more  detailed  information,  such  as  the  algorithms  needed  to  perform  the 
aircraft  tracking  function  and  how  that  function  relates  to  other  functions  in  the  program. 
Programs  are  then  delivered  to  a  configuration  management  group  that  maintains  a  library  of 
program  interfaces  and  functions.  From  there,  the  programs  go  to  a.i  internal  testing  departmen', 
followed  by  an  external  testing  group.  After  passing  the  various  system  tests  the  programs  are 
delivered  and  installed  for  the  fleet  user. 


COLRSE  CONTENT 


The  course  was  designed  so  that  students  could  use  data  relevant  to  FCDSSA  as  they 
learned  about  structured  problem  solving  and  the  basic  graphic  methods.  The  course  began  with 
a  review  of  the  basic  principles  of  TQL,  an  overview  of  the  PDCA  cycle,  and  an  introduction  to 
the  seven  graphic  methods.  Each  graphic  method  was  discussed  within  the  context  of  the  PDC.A 
cycle.  Next,  students  learned  about  the  TQL  Process  Improvement  Model  (PIM),  developed  by 
Houston  and  Dockstader  (1988).  This  model  is  an  elaboration  of  PDCA  activities  and  is 
presented  in  Figure  1  in  the  fomi  of  a  flow  chan.  The  following  sections  are  a  description  of 
activities  conducted  in  each  phase  of  the  PDCA  cycle  and  what  was  done  relevant  to  the  ACDS 
Block  0  process. 


Plan  Phase 


The  first  phase  of  the  PDC.A  cycle  is  the  Plan  phase.  Dunng  the  Plan  phase  critical 
product  and  service  requirements  of  major  customers  are  identified.  Flow'  charts  and  socio- 
technical  svstem  anahsis  are  two  tools  useful  during  this  phase. 

Initial  activities  for  the  students  included  a  discussion  of  Block  0  process  goals  and 
objectives  and  the  identification  of  customers  (both  sponsors  and  fleet  users).  Then,  using  a 
"Nominal  Group  Technique "  (a  variation  of  "brainstorming"),  the  instructor  worked  with  the 
students  to  develop  a  list  of  fleet  user  product  requirements.  Because  process  improvement 
efforts  are  ba.sed  on  these  critical  customer  requirements,  ifs  always  important  to  obtain  this 
information  directly  from  the  customers.  However,  in  this  exercise,  that  was  not  possible  because 
the  class  did  not  have  direct  access  to  the  customers.  The  customer  requirements  of  the  products 
and  serv  ices  from  the  .ACDS  Block  0  process  generated  by  the  class  are  presented  in  Table  2. 
Examples  of  customer  requirements  include:  (1 )  user  fnendliness,  (2)  program  functionality,  (3) 
program  reliability.  (4)  user  support,  and  (5)  availability  of  training.  "Validation  of  these 
requirements  by  the  fleet  customer  was  not  done  as  part  of  this  course  for  the  reasons  died 
above.  However,  the  Department  Director  QMB  is  currently  developing  a  customer  survey 
instrument  to  obtain  information  regarding  the  fleet  customers'  perceptions  of  the  products  and 
service  thev  receive. 


Figure  I.  Process  improvement  model  for  total  quality  management. 


Table  2 


Customer  Requirements  for  Products/Services  for 
Advanced  Combat  Direction  System  (ACDS)  Block  0  Process 
(Results  of  Nominal  Group  Technique) 


Training 

-  Quantity 

-  Simulator 

-  Lock  step 

User  Friendliness  (Makes  job  easier) 

-  Logical  flow  of  operator  functions 

-  Useful  system  tools 

-  Good  human  engineering 

-  Flexibility  (e.g..  adjustable  to  user  experience) 

-  Tailored  (customized  displays) 

-  Consistency  of  procedures 

Functionality  (Meets  or  exceeds  stated  user  requirements) 

-  Adequate  data  links 

-  Effective  direction  of  combat  in  complex  environments 

-  Robust  programs 

-  Programs  work  in  concen  with  manuals 

-  Effective  system  tools  for  job 

-  Accurate  real  world  mapping  displays 

-  Casualty  mode  (graceful  degradation) 

-  Flexible  (customized  displays) 

-  Accurate  real-time  displai  s 

Reliability 

-  Accurate/Consistent  data 

-  Durable  programs  (Large  mean-time-between-failure) 

-  Adaptable  programs  (graceful  degradation) 

-  Diagnostic  capabilities  (trouble-shooting  by  operator) 

-  Accurate  real-time  displays 


Support 

-  Availability  of  resources  (software,  hardware.supplies) 

-  Assistance  (user  support) 

-  Maintenance 

-  Diagnostic  tools 


Table  2  (Continued) 


Documentation 

-  Accurate  manuals 

-  Record  of  program  fixes 

Technology 

-  Fast  program  updates 

-  State-of-the-art  systems 

-  Open  architecture 

Feedback 

-  Opportunity  for  input  for  future  capabilities 

-  Organization  as  user  advocate 

-  Feedback  acknowledged  by  a  real  person 

Configuration  Control 

-  Consistent  procedures  for  changes  and  updates 

Cost 


-  Life  cycle  costs 

-  Low  spaceAVeight  devices 


Delivery 


-Timeliness 


In  the  context  of  the  ACDS,  the  sponsor  can  also  be  thought  of  as  a  customer.  Therefore, 
prod uct/ser\' ice  quality  characteristics  for  the  sponsor  were  also  generated,  this  time  using  a 
more  traditional  brainstorming  session.  The  results  of  that  group  process  are  presented  in  Table 

3.  Following  this  brainstorming  session,  the  group  decided  to  work  on  a  requirement  that  was 
imponant  to  both  the  sponsor  and  fleet  user.  This  requirement  was  the  functionality  of  the 
computer  programs.  Table  4  presents  a  list  of  possible  product  or  output  measures  of 
functionality  for  ACDS  Block  0  programs.  Examples  of  output  measures  for  program 
functionality  include:  (1)  the  number  of  functions  the  program  performs  correctly,  (2)  the  mean 
time  between  program  failures,  and  (3)  the  number  of  times  the  operator  must  intervene  to 
maintain  the  program  in  operation.  This  list  was  generated  using  another  brainstorming  session. 


Table  3 

Sponsor  Product/Service  Requirements 
for  Advanced  Combat  Direction  System  (ACDS)  Block  0  Process 
((Jenerated  from  Brainstorming  Session) 


1.  Cost/Dollars 

2.  Schedule/Timeliness 

3.  Specifications  met 

4.  Reliability/Availahilitv  (s\  stem  doesn't  go  down) 

5.  Availability  of  status  repons 

6.  Maintainability  of  life  cycle  suppon 

7.  Flexibility  of  organization  (change  in  programs,  personnel) 

8.  lnteroperabilit\- 


Table  4 

Product  Measures  of  Functionality 
for  Advanced  Combat  Direction  System  (ACDS)  Block  0  Program 
(Generated  from  Brainstorming  Session) 


1 .  .Number  of  functions  done  correct!)- 

2.  Mean  time  between  failures 

3.  Number  of  times  program  doesn't  function 

4.  Number  of  times  operator  intervenes 

5.  Number  of  trouble  reports 

6.  .Number  of  Beet  messages 

7.  Complexity  of  components 

8.  Number  of  times  to  do  unit  tests 

9.  Length  of  time  program  runs  before  it  fails 


.Another  task  in  the  Plan  phase  is  to  develop  a  process  Bow  chart.  A  flow  chart  is  a 
diagram  that  depicts  the  steps  in  a  process  and  how  these  steps  interrelate.  A  deployment  flow 
chart  (Tribus,  1989)  graphical!)  communicates  the  interrelation  and  sequence  of  operations  and 
the  decisions  required  to  transform  resources  into  outputs.  Figure  2  shows  the  ACDS  Block  0 
process  Bow  chart. 
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Figure  2.  Deployment  flow  chart  of  Advanced  Combat  Direction  System  (ACDS) 
Block  0  process. 
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During  the  discussion  and  development  of  the  process  flow  chart,  the  group  identifitd 
such  things  as  duplicated  efforts  between  operations,  "gaps"  in  accountability,  overuse  ®f 
inspection,  and  ways  to  streamline  the  process.  Table  5  is  a  partial  list  of  ideas  generated  duriEg 
flow  charting  that  were  thought  to  have  a  positive  effect  on  the  ACDS  process. 


Table  5 

Ideas  for  Ideal  Flow  Process 

for  Advanced  Combat  Direction  System  (ACDS)  Block  0  Program 
(Generated  from  Brainstorming  Session) 


1.  Certify  modules 

2.  Make  smaller  modules 

3.  Use  structured  coding 

4.  Show  front-end  concern  for  quality  over  dollars 

5.  Maintain  accurate  documentation 

6.  Finalize  code/Design  walk  through 

7.  Send  analyst  with  delivery  team  to  get  user  feedback 

8.  Fomialize  testing  process 

9.  Fortrt  team  for  up-front  analysis/Design 

10.  Give  Analyst  input  into  testing  procedures 

1 1 .  Reduce  number  of  distractions  (provide  more  .support) 


Although  socio-technical  systems  analysis  is  not  considered  one  of  the  seven  grapfcic 
methods,  it  was  presented  here  as  a  tool  for  understanding  hou  an  organization  transforms  ns 
inputs  into  outputs  through  the  interaction  of  its  social  and  technical  subsystems.  The  technical 
system  was  analyzed  by  identifying  the  unit  operations  and  variances.  Unit  operations  are  Jfie 
activities  required  to  bring  about  a  change  in  the  transformation  process.  Variances  are  tbtnse 
factors  that  create  de\  latioiis  from  some  desired  outcome. 

Table  6  lists  the  "unit  operations"  and  "variances"  for  the  ACDS  Block  0  process.  The 
unit  operations  for  the  Block  0  process  consisted  of  such  things  as  design  of  high-lc^e! 
specifications,  analysis  of  low-level  specifications,  programming,  and  configuration 
management,  while  some  of  the  variances  identified  in  the  Block  0  process  included  clarity  of 
program  performance  specifications  (PPS)  and  correctness  of  analysis  of  high-kvel 
specifications. 

The  next  step  in  a  technical  analysis  is  to  develop  a  variance  matrix.  A  variance  matrix  is 
a  chart  that  lists  all  of  the  variances  for  each  unit  operation.  The  matrix  serv-es  two  purposes:  id ) 
to  show  the  interaction  pattern  of  the  variances  and  (2)  to  help  isolate  key  variances.  A  key 
variance  is  one  that  stops  or  serioush'  disrupts  the  transformation  process.  Figure  3  shows  a 
variance  matrix  for  the  ACDS  Block  0  process,  as  developed  by  the  students.  It  is  incomplete 
because  of  the  lack  of  class  time,  but  it  was  recommended  that  they  complete  it  in  the  fuiine. 
Nevertheless,  from  the  variance  matrix  exercise  as  it  exists,  key  variances  began  to  emerge,  such 
as  clarity  and  accuracy  of  high-level  requirements  and  correctness  of  source  code. 


Table  6 


Unit  Operations  and  Variances  for  Advanced  Combat 
Direction  System  (ACDS)  Block  0  Process 


Unit  Operations  Variances 


Statement  of  high-level  requirements 

Analysis  of  low-level  specifications 

Programming  source  code 


Configuration  management 

Internal  testing 

External  testing 
Dcliverv 


(1)  Completeness 

(2)  Accuracy 

(3)  Clarity 

(4)  Volume  (workload) 

(5)  Timeliness 

(6)  Moving  targets 

(1)  Correctness  of  analysis  of 

high-level  requirements 

(2)  Clarity  of  PPS  (pseudo  code) 

(3)  Timeliness 

(4)  Quality  of  operator  manuals 

(5)  Currency  of  operator  manuals 

(6)  Completeness  of  PPS 

( 1 )  Interpretation  of  pseudo  code 

(2)  Correctness  of  source  code 

(3)  Availability  of  unit  testing 

facilities 

(4)  Frequency  of  spec  changes 

(5)  Currency  of  master  (library’) 

program 

(6)  Knowledge  of  hardware 

(7)  Knowledge  of  interfaces 

(8)  Source  update  cycle 

(9)  Currency  of  programming 

tools 

(10)  Timeliness 

( 1 )  Currency  of  documentation 

(2)  Accuracy  of  TR  data  base 

(3)  Appropriateness  of  TR  priority 

(4)  Adequacy  of  compiling 

procedures 

( 1 )  Quality  of  test  facilities 

(2)  Availability  of  test  facilities 

(3)  Adequacy  of  simulations 

(4)  Interpretation  of  PPS 

( 1 )  Know  ledge  of  fleet 

requirements 

(2)  Interpretation  of  PPS 

(1 )  Availability  of  transport 

(2)  Delivery  team  mix 


Note.  PPS  =  Program  Perfonnance  Specifications.  TR  =  Trouble  Reports. 
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Figure  3.  Partial  variance  matrix  for  Advanced  Combat  Direction  System  (ACDS) 
Block  0  process. 


The  next  step  was  to  construct  a  table  identifying  the  key  variances  and  the  negative 
impact  the  variances  have  on  the  achievement  of  goals,  the  frequency  of  their  occurrence,  and 
their  impact  on  output  (Table  7).  A  variance  control  table  was  then  constructed  that  listed  the  key 
variances  and  documented  current  control  mechanisms  and  preliminary  ideas  for  improvement 
(Table  8). 


Do  Phase 


After  quality  goals  have  been  defined,  the  process  variables  related  to  improved  quality 
need  to  be  identified.  There  are  three  major  responsibilities  for  the  team  in  the  Do  phase.  First,  it 
studies  the  current  process  and  its  outputs  to  identify  variables  related  to  quality.  Second,  it 
identifies  measures  of  those  variables.  Third,  it  creates  a  format  by  which  to  record  and  collect 
data. 


The  purpose  of  conducting  the  cause-and-effect  analysis  is  to  identify  the  variables  that 
appear  to  have  major  influence  on  process  results.  Cause-and-effect  diagrams  show  the 
relationship  between  a  given  "effect"  and  the  possible  "causes "  that  contribute  to  it.  Cause-and- 
effect  diagrams  are  used  for  analyzing  problems  and  the  factors  that  contribute  to  them.  A  cause- 
and-effect  diagram  was  developed  using  a  product  quality  characteristic  (program  functionality) 
that  was  considered  to  be  important  to  both  the  sponsor  and  the  fleet  user.  Figure  4  shows  the 
results  from  the  group  brainstorming  session.  As  can  be  seen  in  Figure  4.  softw-are  tools, 
hardware,  personnel,  management  support,  methods,  and  quality  evaluation  were  identified  as 
important  "causes "  of  process  performance.  Functionality  of  the  computer  programs  was  the 
result  or  "  effect"'  of  the  combination  of  variables  or  "causes." 

As  important  as  it  is  to  have  valid  measures  of  outcomes  (customer  satisfaction-customer 
requirements)  and  outputs  (products/services),  it  is  vital  to  obtain  process  measures  as  well. 
Process  measures  are  measures  that  reflect  the  optimization  of  an  organization's  internal 
processes.  Precise  process  measures  are  critical  because  they  have  a  direct  effect  on  output 
quality.  Examples  of  process  measures  for  software  development  might  include  (1)  the  number 
of  lines  of  code  per  program  specification.  (2)  the  amount  of  time  spent  clarifying  the 
specification.  (3)  the  number  of  unit  tests  per  lines  of  code,  or  (4)  machine  downtime. 
Unfortunately,  organizations  rarely  have  systems  established  to  collect  data  on  process 
chiu'acteristics.  When  such  data  are  not  available,  it  is  necessary  to  develop  those  process 
measures.  Developing  process  measures  is  not  easy.  .Although  the  group  discussed  possible 
measures,  they  were  unable  to  come  to  any  agreement  on  a  set  of  them  that  would  adequately 
reflect  the  ACDS  Block  0  process. 


Table  7 

Identification  of  Key  Variances 
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Table  8 

Variance  Control  Table 
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Figure  4,  Cause-and-effect  diagram  for  functionality  of  Advanced  Combat 
Direction  System  (ACDS). 


Once  process  measures  are  developed,  the  team  must  decide  how  to  collect  the  data.  The 
instructors  then  oresented  material  on  data  collection  purposes  and  procedures.  Types  of  data, 
sampling  techniques,  and  the  use  of  check  sheets  were  discussed.  The  first  part  of  the  data 
collection  strategy  requires  the  team  to  collect  information  on  the  "causes”  of  variation  identified 
through  the  cause-and-effect  analysis.  Questions  concerning  whai  infomiation  will  be  collected, 
how  it  will  be  collected,  who  will  collect  it,  where  it  will  be  collected  and  when  it  will  be 
collected  need  to  be  answered.  Then,  using  a  Pareto  diagram,  the  most  important  causes  can  be 
identified. 

Pareto  charts  provide  a  basis  for  selecting  "problems"  on  which  to  focus  initially  and 
indicate  results  of  improvement  efforts  (baseline  vice  results  of  changes).  A  Pareto  chart  is  a 
vertical  bar  graph  that  shows  how  categories  are  compared  and  ranked  (see  Figures  5,  6,  A-l, 
and  A-2  for  e.xamples).  The  bars  are  arranged  in  descending  order  of  magnitude  from  left  to 
right,  and  often  a  line  is  drawn  from  left  to  right  that  show's  the  cumulative  frequency  (see 
Figures  5  and  6).  Sometimes  the  categories  in  a  Pareto  chan  need  to  be  redefined  on  the  basis  of 
initial  results.  For  example.  Figure  5  shows  that  the  "other”  and  "document"  categories  are  the 
most  frequent.  The\  are  represented  to  the  right,  however,  to  indicate  that  these  categories 
belong  to  other  categories  or  should  be  broken  down  into  new  categories.  Figure  6  shows  what 
the  Pareto  chart  looks  like  after  the  category  adjustments  are  made. 


Check  Phase 


Typically,  we  tend  to  measure  the  final  product  or  service,  this  is  our  output.  The 
customer  response  or  reaction  to  the  product  or  serx  ice  is  the  outcome.  Outcomes  are  indicators 
of  how  well  the  product  or  service  satisfied  the  customers'  needs.  Process  factors  are  how  such 
things  as  machines,  methods,  material,  and  people  interact  to  produce  the  organiation's  product 
or  serxice.  These  have  been  the  most  overlooked  sources  of  measurement.  Analyzing  what 
occurs  du.  ig  this  phase  allowx  us  to  determine  how  our  products  and  serx  ices  are  produced  and 
how  It  can  be  improxed. 

In  the  Check  phase  teams  collect  process  and  output  data  and  summarize  the  data  using 
graphic  methods.  Once  the  data  have  been  summarized,  teams  can  detemiine  which  process 
variables  have  a  significant  effect  on  outputs  and.  subsequently,  outcomes.  There  arc  five 
graphic  methods  commonly  associated  with  process  analysis:  Pareto  diagrams,  histograms, 
scatter  diagrams,  run  charts  and  control  charts.  Although  these  graphic  methods  are  most 
appropriate  for  process  measures,  product  output  data  already  available  from  the  organizaiaon 
were  used  in  the  class  to  illustrate  their  use  and  interpretation.  The  output  data  was  in  the  form  of 
Trouble  Reports  (TRs). 

When  problems  are  discovered  in  the  programs  generated  by  the  Block  0  process,  a  TR  is 
written.  The  TR  goes  into  a  database  that  can  be  examined  by  program  analysts.  Information  in 
the  database  includes:  a  brief  description  of  the  nature  of  the  problem,  w  ho  found  the  problem, 
w  here  the  problem  originated,  the  priority  for  fixing  the  problem,  and  the  current  status  of  the 
TR.  This  database  w  as  used  to  demonstrate  the  use  of  Pareto  charts,  histograms,  scatter  diagrams 
and  run  charts. 

Figures  5  and  6  and  Tables  9  and  10  show  data  sheets  and  Pareto  charts  created  to 
inx  estigate  w  hich  program  modules  contributed  the  most  TRs.  From  a  review  of  a  Pareto  chart, 
a  team  could  identify  those  variables  that  have  the  greatest  effect  on  an  output  characteristic. 
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Figure  5.  Pareto  chart  of  software  modules. 
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Figure  6.  Pareto  chart  of  top  seven  modules. 
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Data  Sheet  for  Pareto  Chari  (liieliirles  Modules  for  all  Trouble  Reports) 
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Data  Sheet  f<ir  Paret«>  Chart  (Top  Seven  Modules) 
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Histograms  are  used  to  characterize  variables  that  are  measured  and  provide  a  snapshot 
of  the  variation  in  the  measured  characteristic.  Histograms  can  be  used  to  depict  variation  in 
process  performance  or  results.  A  histogram  is  a  bar  graph  that  summarizes  a  large  set  of  data 
along  a  continuum.  Figure  7  presents  a  histogram  that  depicts  the  frequency  of  occurrence  of 
TRs  of  different  oriorities  (most  critical  or  severe).  The  chart  shows  that  TRs  with  pnority  of  3 
or  4  are  the  most  frequent. 
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CONTINUOUS  VARIABLE  (TR  PRIORITY) 


Figure  7.  Histogram  of  TR  priority. 

Scatter  diagrams  are  often  used  to  check  the  strength  of  the  possible  "cause-and-effect" 
relationships  identified  in  the  Do  phase.  Scatter  diagrams  show  the  relationship  between  two 
variables  that  have  been  paired  on  some  basis.  The  shape  of  the  pairing  of  points  in  the  scatter 
diagram  (often  a  statistical  procedure  is  used  to  derive  a  "line  of  best  fit”)  indicates  whether  the 
two  variables  are  related.  Figures  8  and  9  depict  scatter  plots  (scatter  diagrams)  of  TRs.  Figure  8 
shows  the  relationship  between  TR  priority  and  days-in-status  for  TRs  generated  from  internal 
testing  groups,  while  Figure  9  displays  the  same  relationship  for  TRs  generated  by  external 
testing  groups. 


A  run  chan  is  shown  in  Figure  10.  Run  charts  are  constructed  to  determine  if  there  are 
time-related  patterns  in  process  performance.  They  can  also  be  used  to  test  "before"  and  "after" 
effects  of  process  changes.  Figure  10  presents  the  number  of  TRs  generated  for  an  eleven  day 
stretch. 


Control  charts  depict  process  perfomiance  from  samples  taken  over  a  period  of  time. 
They  can  be  used  to  predict  how  a  process  should  perform  under  stable  conditions.  These  charts 
can  be  used  to  distinguish  between  variables  that  consistently  affect  all  of  the  outputs  of  a 
process  (called  "common  causes")  and  those  that  affect  outputs  in  an  unpredictable  way  (called 
"special  causes").  Control  charts  were  discussed  following  the  viewing  of  a  film  titled  "A 
Japanese  Control  Chart".  Panicipants  then  constructed  X  bar  and  R  charts  using  hypothetical 
data.  Because  we  did  not  have  access  to  a  sufficient  amount  of  data  to  construct  an  actual  control 
chart  on  the  .ACDS  Block  0  process,  the  panicipants  constructed  control  charts  using  sample  data 
from  an  in-class  exercise  (Figure  1 1 ). 


DAYS  IN-STATUS 


PRIORITY 


Figure  8.  Scatter  plot  of  priority  and  days-in-status  for  internal  Trouble  Reports. 


PRIORITY 

Figure  9.  Scatter  plot  of  priority  and  days-in-status  for  external  Trouble  Reports. 
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^  NUMBKR  OF  TROUBLE  RFJ>()RTS 


30 


Figure  10.  Run  chart  of  TRs. 


Sole  L'CL  =  I'ppi’r  Control  Limn.  LCL  =  Loner  Control  Limit 

Figure  11.  Control  charts  using  hypothetical  data. 
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Act  Phase 


At  the  conclusion  of  the  Check  phase,  teams  are  supposed  to  select  process  variables 
believed  to  be  major  contributors  to  process  quality  and  take  action.  During  this  exercise  one 
major  problem  with  the  Block  0  process  became  obvious  to  the  group.  It  was  the  lack  of  a  design 
document.  The  group  decided  to  investigate  this  part  of  the  process  more  completely.  Three 
groups  were  formed,  each  one  looking  at  a  slightly  different  aspect  of  the  problem.  After 
reviewing  the  importance  of  data  collection  (including  hypothesis  generation,  types  of  data, 
sampling,  and  check  sheet  development),  each  group  developed  a  data  collection  plan  (Table 
11).  After  approximately  2  months  of  data  collection,  the  groups  presented  the  results 
graphically  and  provided  recommendations  to  the  class  for  process  improvement  (Appendix). 


Table  11 

.Summary  of  Things  to  Consider  in  Data  Collection: 
What  do  We  Need  to  Know? 


1 .  Purpose  of  data  collection. 

2.  What  will  be  done  with  the  data'i’ 

a.  Feedback 

b.  Greater  understanding 

c.  Action 

Types  and  sources  of  infomiation  that  need  to  be  gathered  (i.e..  frequencies,  individual, 
group,  other  agencies,  e.xpensy 

4  .Methods  of  gathering  data  (i.e..  inten  iews.  checksheets,  e.xisiing  data)  and  the  format 
and  structure  for  each. 

."i.  .Sampling  plan  (i.e..  when  and  how  often  data  will  be  recorded  and  reported). 

6.  Who  w  ill  collect  the  data? 

7.  How  will  the  data  be  summarized  and  represented?  Which  graphic- 

tool  w  ill  be  used'.’ 


LESSONS  LEARNED 

Perceptions  of  the  Course  by  the  Participants. 

The  participants  reported  that  communication,  especially  interdepartmental,  had 
improved  as  a  result  of  the  course  and  the  group  discussions.  Participants  from  the  different 
codes  got  an  opportunits  to  see  that  the  computer  program  generation  process  had  the  same 
kinds  of  problems  regardless  of  the  program  platform  or  function.  The  development  of  a  team 
invoking  the  fleet  user,  the  sponsors,  and  FCDSSA  was  often  mentioned  as  a  way  to 
communicate  accurately  customer  requirements  and  specifications. 


The  participants  also  reported  that  use  of  a  real  FCDSSA  process  facilitated  their 
understanding  of  the  uses  and  interpretation  of  the  graphs  and  charts.  They  also  believed  they 
were  more  involved  than  they  would  have  been  had  the  process  been  hypothetical.  It  was 
difficult,  however,  to  cover  the  activities  required  for  structured  problem  solving  and  use  of  the 
tools  in  sufficient  depth. 


Perceptions  of  Course  by  the  Instructors 

Time  was  a  problem.  Process  analysis  and  improvement  can  take  many  months,  yet 
content  and  activities  had  to  be  covered  in  2  weeks.  A  second  problem  concerned  the 
development  of  process  measures.  The  development  of  operational  definitions  is  critical  to  this 
measurement  task.  Participants  found  that  identifying  measures  and  measurement  itself  to  be 
very  difficult.  Another  problem  was  the  temptation  to  use  existing  data  to  answer  questions.  This 
can  lead  to  different  interpretations  of  the  same  data.  Discussions  surrounded  the  need  and 
importance  of  developing  unique  measures  to  address  specific  hypotheses.  It  also  appeared  as 
though  participants  were  skeptical  about  how  the  measures  would  be  used  as  well  as  whether  the 
measures  developed  would  reflect  process  performance  or  their  job  performance. 

The  presentation  sequence  of  the  seven  graphic  tools  here  generally  reflected  their  order 
and  use  in  process  improvement  activities.  However,  this  is  not  the  only  order  in  which  they  can 
be  used.  Tools  can  be  used  during  any  part  of  the  PDCA  cycle  depending  on  the  question  the 
group  wants  answered. 

Finally,  the  major  impediment  to  the  use  of  TQL  and  process  improvement  is  not  likely 
to  be  the  nature  of  the  process  investigated  or  the  difficulty  of  identifying  measures,  but  rather 
the  attitudes  and  practices  of  managers.  Managers  need  to  understand  and  agree  that  the  TQL 
approach  can  be  used  to  improve  significantly  the  products  and  services  of  their  organization. 


RECOMMENDATIONS 

1 .  Organizations  interested  in  TQL  should  provide  training  to  their  employees  on  the  graphic 
methods  for  quality  improvement.  The  course  should  be  taught  by  trained  personnel  and  should 
meet  on  a  weekly  or  biweekly  basis  for  several  months. 

2.  The  graphic  methods  can  be  learned  effectively  by  applying  these  tools  to  a  real  process  that 
has  significance  to  the  organization.  The  ESC  (or  similar  body  of  top  managers)  should  be 
responsible  for  identifying  the  process. 

3.  The  training  should  cover  all  of  the  graphic  methods  that  an  organization  is  likely  to  need. 
The  training  should  demonstrate  how  these  tools  and  techniques  can  be  applied  at  several 
different  phases  of  the  PDCA  cycle,  and  are  not  restricted  to  only  a  single  phase. 

4.  Training  should  be  extended  over  several  months  so  that  long  term  data  can  be  collected. 
Control  charts  should  be  used  to  determine  the  stability  of  the  process.  If  the  process  is  not 
stable,  an  attempt  should  be  made  to  stabilize  it,  if  possible,  before  applying  other  charts  and 
computations. 
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APPENDIX 

GROUP  DATA  COLLECTION  PLANS 


A-0 


(jToup  Data  Collection  Plans 


Each  group  answered  the  items  regarding  data  collection  shown  in  Table  11.  The 
summary  provided  focus  and  structure  to  the  group  discussions. 


Group  1  investigated  the  various  reasons  why  Trouble  Repons  (TRs)  are  written  and  the 
relative  imponance  of  each,  TRs  are  written  for  a  variety  of  reasons.  A  Program  Performance 
Specification  (PPS)  may  be  too  general,  a  Program  Design  Specification  (PDS)  may  be  lacking, 
and  the  relationship  between  PPS  and  the  code  may  be  incorrect.  Hopefully  the  data  will 
indicate  whether  designers  and  analysts  need  more  detailed  specifications  and/or  whether 
designer  and  programmer  understand  the  requirements.  The  group  felt  that  this  part  of  the 
process  was  responsible  for  most  of  the  TRs.  The  data  consisted  of  expen  judgments  about 
approximately  100  TRs.  Only  TRs  that  were  "closed”  or  "fixed"  were  used.  The  group 
developed  and  defined  a  classification  scheme  that  categorized  problems  that  possibly  generated 
each  TR.  The  categories  included  (1 )  high-level  coding,  (2)  mistakes  in  the  PPS,  (3)  using  the 
wrong  system  operator  manual  (SOM).  (4)  mistakes  in  coding  (5)  using  the  wrong  te-st 
procedure.  (6)  mistakes  made  b\  the  tester,  (7)  requirements  not  specified,  (8)  mistakes  in  the 
Interface  Design  Specifications  (IDS).  (9)  problems  w'ith  hardware,  (10)  mismatch  between 
coding  and  design,  and  (11)  problems  w  ith  configuration  management,  TRs  were  classified  and 
a  Pareto  diagram  developed  to  communicate  the  results. 


The  goals  and  objectives  of  the  second  group  were  similar  to  those  of  the  first  group.  The 
group  was  interested  in  identifying  the  possible  causes  of  coding  problems.  TRs  identified  by 
Group  1  as  having  coding  problems  were  looked  at  by  Group  2  in  more  detail.  The  categories 
they  used  for  classifying  TRs  w  ith  coding  problems  included  unclear  PPS,  error  on  the  PPS,  or 
incomplete  PPS.  Both  groups  hoped  to  document  the  need  for  a  lower  level-design  document. 


Groups  1  and  2  concluded  frtrin  the  Pareto  chan  (Figure  .A-1 1  that  there  was  something 
wrong  in  the  coding  of  the  software.  The  first  four  categories  on  the  Pareto  chan  involved  coding 
mistakes  due  to  lack  of  specification,  mistakes  in  PPS,  or  mistake  in  the  SOM.  The  groups 
hypothesized  that  problems  in  design  documents  were  caused  by  inadequate  documents  and 
programming  code.  The\  recommended  further  >iudy  to  determine  the  causes  of  inaccurate 
coding;  correction  would  hopefulls  reduce  the  number  of  TRs  produced.  They  recommended 
investigating  low-level  (machine)  and  high-level  (compiler)  programming  codes. 


Group  3  investigated  whether  excessive  mone\,  time,  and  personnel  were  being  used 
analyzing  invalid  external  TRs.  They  hoped  to  determine  the  price  of  nonconformance.  The 
group  s  effort  was  described  as  exploratory  in  nature.  They  looked  at  existing  TRs  in  the 
database  that  had  a  status  code  of  Reject  as  Invalid.  Fixed  (RIF).  They  counted  the  number  of 
TRs  generated  by  the  external  agency  relative  to  the  total  (Figure  A-2).  They  concluded  that  the 
generation  of  external  TRs  wasn't  as  big  a  problem  as  they  thought.  They  did  report,  however, 
that  they  thought  the  amount  of  time  spent  reviewing  TRs  was  dependent  on  the  level  of 
management  involved.  The  group  felt  that  the  information  and  what  they  learned  from  this  effort 
would  be  useful  to  Fleet  (fombat  Direction  Systems  Suppon  Activity  (FCDSSA)  because  it 
would  help  build  communication  between  FCDSSA  and  the  external  agency. 
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Figure  A-1.  Pareto  chart  of  root  causes  of  TRs. 
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Status  Codes 

Figure  A-2.  Number  of  external  TRs  with  status  codes. 
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